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l.  introduction 

W  A  development  program  can  be  described  and  evaluated  by 
three  interrelated  parameters:  cost,  schedule,  and  technical 

performance.  These  parameters  can  be  visualized  as  the  x,  y,  and  z 
axes  of  a  three-dimensional  space.  A  program's  desired  objective, 
usually  represented  by  a  conceptual  design,  is  a  point  in  the  three- 
dimensional  space  defined  by  the  estimated  or  target  values  for 
cost,  schedule,  and  desired  technical  performance  of  this  conceptual 
design.  To  reach  this  goal,  a  series  of  events  is  laid  out  which  defines 
the  development  path.  Each  event  contains  its  own  cost,  schedule, 
and  technical  performance  characteristics.  Ideally,  the  summation  of 
the  individual  event  parameters  equals  the  whole.  It  is 
impracticable,  if  not  impossible,  to  condense  (most  programs  to  a 
single  cost,  schedule,  and  technical  performance  parameter. 
However,  it  is  often  possible  to  condense^  myriad  development 
activities  into  a  series  of  key  activities  and  events  which  provide  a 
reasonably  accurate  depiction  of  the  overall  program. 

Because  of  the  uncertainty  of  predicting  the  outcome  of  future 
events,  almost  from  the  moment  a  program  is  baselined, 
programmatic  forces  tend  to  impose  changes  on  the  baseline 
parameters.  This  is  especially  true  of  research  and  development 
programs,  like  the  Strategic  Defense  Initiative,  where  many  projects 
are  pushing  the  frontiers  of  science  and  technology.  However,  all  but 
the  purest  of  R&D  programs  have  a  vision  of  the  desired  product;  for 
SDIO-it  is  a  ballistic  missile  defense  system.  Although  there  are 
usually  many  paths  to  reaching  this  end  product,  there  are  not  an 
infinite  number.  Since  changes  to  the  baseline  program  are 
practically  inevitable,  the  challenge  to  the  program  manager  is  to 
ensure  that  the  program  remains  on  a  path  leading  to  the  desired 
product. 

In  practice,  changes  force  the  program  manager  to  examine 
trade-offs  involving  the  three  basic  program  parameters.  In  order  to 
perform  effective  trade-off  analyses,  the  manager  must  understand 
how  each  alternative  impacts  planned  development.  Much  effort  has 
been  invested  in  tracking  and  analyzing  cost  and  schedule 
parameters.  This  is  evidenced  by  the  DODD  7000  series  directives 
requiring  cost  and  schedule  tracking  systems  for  major  Government 
contracts.  Cost  and  schedule  management  has  it’s  own  set  of 
terminology  (i.e.,  Budgeted  Cost  of  Work  Performed,  Actual  Cost  of 
Work  Performed,  Estimate  To  Complete,  etc.)  Unfortunately, 


technical  progress  reporting  has  not  reached  the  same  level  of 
standardization,  automation,  or  definition. 

The  RRI  team  approach  to  evaluating  SDIO  technology 
performance  proposes  condensing  the  technical  status  of  individual 
programs  to  a  manageable  number  of  categories  and  assessing  the 
status  using  a  standard  framework.  This  is  accomplished  by  building 
on  the  technology  maturity  assessment  work  previously  performed 
within  SDIO.  This  methodology  has  the  additional  advantage  of 
allowing  for  a  more  uniform  integration  of  technology  data  with 
associated  cost  and  schedule  data. 

2.  History  of  SDJO  Technical  Maturity  Assessment 

SDIO  has  attempted  to  define  technical  maturity  before. 
Borrowing  from  NASA  and  the  aerospace  industry,  technical  maturity 
was  divided  into  ten  levels  progressing  from  an  initial  conceptual 
stage  to  deployment.  The  problems  with  this  approach  were, 
generally  .threefold.  F’.rst,  it  was  a  standalone  depiction  that  did  not 
tie  maturity  to  a  well  defined  system  development  framework. 
Second,  the  levels  were  defined  at  a  very  general  level  and  could  not 
be  readily  applied  to  the  diversity  of  programs  within  SDIO.  Third, 
program  managers  were  left  on  their  own  to  determine  exactly  how 
their  program  fit  into  the  maturity  levels.  This  created  a  large 
variety  of  interpretations  of  where  program  events  fell  on  the 
maturity  scale. 

The  RRI  team  approach  to  addressing  these  shortfalls  is  to  use 
the  DOD  System  Lifecycle  Development  process,  or  DAB  process,  as 
the  framework  for  SDS  development.  The  basic  steps  of  our 
approach  are  summarized  below: 

Map  the  existing  technical  maturity  level  definitions  into  the 
DAB  process 

Expand  those  definitions  from  the  top  down  and  from  the 

bottom  up.  Top  down  expansion  ensures  that  the  definitions  are 
appropriately  influenced  by  the  presence  of  key  policy  and  decision 
making  issues.  Bottom  up  expansion  ensures  that  SDIO  program 

managers  can  effectively  map  their  program  events  into  the 
definitions. 

Map  existing  data  (e.g.  from  the  Program  Master  Plan)  on 

individual  program  development  issues  into  the  appropriate 
maturity  level.  The  issues  should  be  traceable  to  system 

requirements  and  expressed  in  engineering  units  of  the  required 
technical  parameters  (e.g.  megawatts,  km/sec,  micro-radians,  etc). 

Use  the  program  control  personnel  in  each  of  the  deputates  to 
assist  the  program  managers  in  the  mapping  process.  This  will 


ensure  consistency  in  interpretation.  PMA  tasks  should  focus  on 
efforts  that  increase  technical  maturity. 

Perform  project  risk  analysis  in  the  context  of  the  cost, 
schedule  and  technical  risks  associated  with  progression  from  one 
level  of  maturity  to  the  next  within  the  given  budget,  schedule,  and 
existing  project  configuration. 

Refine  overall  connectivity  as  the  dependencies  and  barriers 
for  moving  from  one  level  to  the  next  become  apparent. 

2.1  Mapping  Technical  Maturity  Levels  Into  The  DAB 
POfcfe&S 

Table  1  depicts  the  mapping  of  the  technical  maturity  levels 
into  the  DAB  process. 

2.2  Expand  the  Definitions 

The  DAB  process  is  the  DOD-accepted  structure  for  systems 
development  and  acquisition.  However,  due  to  SDIO's  political 
visibility  other  factors  beside  the  typical  acquisition  issues  influence 
the  program.  Conceptually,  this  effect  is  illustrated  in  Figure  1.  The 
pyramid  represents  the  presence  of  a  few  key  policy  level  issues  and 
directives  that  flow  down  to  the  program  while  the  more  numerous 

program-specific  development  issues  flow  up.  The  SDS  program  lies 
in  the  middle  and  the  SDIO  Director  and  staff  work  to  balance  the 
development  program  to  ensure  that  the  top  down  issues  are 
adequately  addressed  and  accomodated  with  full  consideration  of 
the  reality  of  the  technical  issues  that  flow  upward. 

Time  is  a  common  factor  to  these  processes.  For  example, 

many  of  the  top  down  directives  are  tied  to  dates,  e.g.,  a  Presidential 
decision  in  1992/1993,  deployment  timeframe,  ABM  Treaty 
compliance  until  199X,  etc.  The  bottom  up  techni>  i  development 
issues  are  tied  to  time  by  the  reality  of  what  can  be  c  aplished  in 
a  given  period  with  a  given  budget.  Finally,  the  DAd  process  is 
basically  a  chronological  development  methodology. 

Figure  2  integrates  the  top  down  and  bottom  up  issues,  the  DAB 
framework,  the  technical  maturity  levels  of  the  assorted 
technologies,  and  the  time  reference.  The  technical  maturity  levels, 
influenced  by  the  top  down  directives,  define  the  required  progress 

necessary  for  each  individual  program.  Management  decisions  can 
then  be  made  based  on  the  relative  maturity  levels.  For  example, 

accelerating  or  decelerating  a  program  (via  budget  actions)  by 
comparing  its  current  maturity  !evel  with  the  desired  level. 

Expanding  the  technology  maturity  definitions  is  crucial  fo  the 
process.  There  are  several  efforts  underway  to  develop  decision 


Table  1 :  Mapping  Technology 
Maturity  Levels  Into  the  DAB  Process 
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Deployment  1 0  -  System  Deployment  1 0  -  Performance  of  the  first  article  acceptance  testing 

and  deploy  for  operational  utilization 


Figure  t :  The  SDIO  Program  Is  Pulled  By  Top 
Down  And  Bottom  Up  Issues 
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Figure  2:  Integrating  Policy,  DAB,  And 
Technology  Development  Issues 


UJ 

5  / 

> 

is 

Cv  <7 

7 

O 

CD 

^  v-  C/) 

<  o>  . 

/ 

_i 

o  o> 

/ 

o 

Ol 

< 

Q 

CL 

<  LU 
2  _J 


£  _  t  e 

q-  oo  c  j  2l  iu  £ 

«! 


S  S' 


criteria  for  various  aspects  of  the  program.  Some  address  DAB 
Milestone  II  criteria,  others  NSD  14  and  the  1992/1993  Presidential 
decision.  All  these  efforts  can  be  valuable  but  they  represent  only 
single  points  in  time.  Key  development  decisions  are  based  not  only 
on  current  status  but  on  the  relative  merits  of  proceeding  to  the  next 
decision  point  or  milestone  eventually  leading  to  the  ultimate  goal. 
All  of  these  point-in-time  decision  criteria  efforts  should  be 
integrated  into  the  DAB  development  framework  and  checked  from 
time  to  time  for  continued  consistency  and  realistic  progression 
toward  the  overall  goals.  Finally,  the  maturity  level  definitions  must 
be  usable  by  all  program  managers  and  with  minimal  variation  in 
interpretation . 

The  use  of  simulation  is  one  example  of  the  need  to  expand  on 
the  definitions  of  technical  maturity  levels.  More  than  perhaps  any 
other  past  development  program,  SDS  will  depend  on  simulation  and 
analysis,  vice  actual  developmental  and  operational  testing,  to 
confirm  performance.  What  needs  to  be  proven  by  simulation  and 
the  respective  levels  of  acceptability  must  be  a  part  of  technical 
maturity  assessment.  Figure  3  provides  an  example  of  how 
definitions  could  be  expanded  to  account  for  such  areas  of  increased 
emphasis  as  system  level  simulation. 


1*2 _ Map _ Existing _ P-£ggram  Dmlppmegt _ Issues  into 

Maturity  Levels 

The  Program  Master  Plan  contains  a  list  of  key  issues  for  each 
SDIO  element.  These  issues  should  define  what  must  be  done  in 
order  to  meet  an  element's  system  goals.  They  should  be  the  focus  of 
the  development  thrust.  Each  SDS  element  should  have  a  set  of 
system  Figures  Of  Merit  (FOMs)  or  Measures  of  Effectiveness  (MOE). 
These  FOMs  define  the  actual  performance  of  the  element  and  are 
used  by  the  system  architects  to  define  the  SDS  capabilities. 
Technical  issues  should  be  expressed  in  terms  of  these  FOMs  and 
mapped  into  the  maturity  levels.  Figure  4  illustrates  a  format  for 
technical  performance  culminating  in  the  actual  mission  requirement 
(the  FOM).  This  approach  aids  in  requirements  traceability.  As  a 
system  advances  through  the  maturity  levels,  progress  toward  the 
system  FOM  is  apparent.  This  approach  of  using  actual  parameters 
avoids  schemes  that  attach  levels  of  confidence  or  other  subjective 
numbering  systems  to  issues. 

Once  the  individual  issues  are  quantified,  the  technical 
maturity  of  the  element  as  a  whole  can  then  be  expressed  in  terms  of 
these  issues.  Of  course,  the  maturity  levels  for  the  various  elements 
or  technologies  may  be  quite  different  for  a  given  point  in  time.  A 
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DESIGN,  FABRICATE,  AND  TEST  PROTOTYPE  COMPONENTS  OR  SYSTEMS  IN  A 
RELEVANT  TEST  ENVIRONMENT  WHICH  MAY  CONSIST  OF  HARDWARE-IN  THE 
LOOP  TESTING  WITH  NTB  SYSTEM  LEVEL  SIMULATION  TOOLS 


Figure  4:  Program  Development  Issues  Are  Expressed  In 
Terms  Of  System  Goals  And  Mapped  Into  The  Maturity  Levels 


basic  rule  would  be  that  the  element  is  no  more  mature  than  the 
level  of  its  least  mature  issue.  This  will  alow  an  element  program 
manager  to  realign  resources  to  balance  his  program  in  support  of 
system  level  milestones  or  required  maturity  levels.  At  a  higher 
level,  it  will  allow  the  SDIO  director  to  shift  resources  allocated  to  the 
elements  and  other  parts  of  the  program  as  necessary  to  achieve 
the  desired  maturity  level  of  the  system.  To  minimize  the  diversity 
of  maturity  levels  among  elements  for  the  1992/1993  Presidential 
decision,  some  programs  may  need  to  be  accelerated  while  others  are 
decelerated.  Fig.  5,  for  example,  shows  why  accelerating  Brilliant 
Pebbles  and  decelerating  SBI  may  be  necessary  so  that  both  are  at  a 
comparable  state  of  maturity  in  that  time  frame.  The  maturity 
definitions  provide  guidance  on  how  the  programs  should  be 
structured  to  meet  this  goal  and  provide  a  baseline  to  assess  progress 
and  compliance. 


The  recent  reorganization  of  SDIO  included  plans  for  placing 
Planning  and  Control  personnel  in  each  of  the  Deputates.  These 
personnel  could  be  tasked  with  assisting  the  element  program 
managers  in  using  the  maturity  level  definitions  to  depict  their 
programs.  The  Planning  and  Control  office  should  utilize  their 
assessment  capabilities  to  assess  individual  program  progress  and 
emphasize  efforts  to  advance  the  technical  maturity  of  the  key 
issues.  This  creates  a  direct  tie  to  budget  and  schedule.  Figure  6 
illustrates  a  step-by-step  process  for  using  technical  maturity. 


Connectivit 


ritical  Path  Analvsi 


and  Risk 


By  breaking  the  key  issues  into  technical  maturity  levels,  the 
element  program  manager  can  structure  development  in  a  sequential 
and  evolving  manner.  This  should  yield  a  connectivity  model  that 
can  be  integrated  directly  into  the  overall  SDS  connectivity  model. 
The  relative  maturity  of  one  program  to  others  as  a  function  of  time 
can  be  displayed.  Decisions  can  be  made  concerning  the  allocation  of 
resources  to  address  shortfalls. 

Risk  assessment  aids  the  decision  making  process  because  it 
can  be  performed  in  the  context  of  evaluating  progress  through  the 
maturity  levels.  The  maturity  levels  would  identify  which  issues 
were  being  addressed  and  exactly  what  kind  of  technological 
progress  was  required  (e.g.  moving  from  level  3  to  4  in  the  next  FY). 
The  PMA  and  connectivity  data  would  describe  the  schedule  and 
resources  allocated.  Risk  assessment  would  examine  the  expected 


Figure  5:  Balancing  The  Program  Elements  Using 

Technology  Maturity 
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technical  progress  given  the  allocated  resources  and  time  and 
evaluate  the  probability  of  actually  making  that  progress.  (The  exact 
risk  assessment  methodologies  will  be  discussed  in  another  paper.  ) 


& _ Summary 

The  effective  use  of  the  technical  maturity  methodology 
requires  an  overall  system  development  structure  or  framework 
which  the  DAB  process  fulfills.  Existing  efforts  to  define  decision 
criteria  for  program  management  guidance  are  valuable  but  do  not 
go  far  enough.  They  must  be  used  as  input  to  the  technical  maturity 
level  assessment  methodology.  Additional  issues,  such  as  the  degree 
of  simulation  required,  must  be  addressed  in  expanded  definitions  of 
the  technical  maturity  levels  The  process  should  be  applied 
uniformly  throughout  SDIO  which  should  be  possible  in  light  of  the 
planned  allocation  of  Planning  and  Control  office  personnel  to  the 
deputates.  These  efforts  mesh  well  with  risk  analysis  and  critical 
path  development  work  now  ongoing  which  altogether  create  an 
integrated  program  model  that  will  provide  decision  makers  with  the 
data  and  confidence  they  need  to  direct  this  very  complex  program. 
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